I think I finally got everything working. It turns out that simply calling git submodule update --init triggered 2 distinct problems:

1. sometimes the submodule would not be in the proper state (this problem comes up even when we disable git-lfs)
2. sometimes git-lfs just fails

I have cases where problem 2 occurs without problem 1 and I have seen cases where both problems occur.

We adopt the following work-around:

1. we invoke git submodule update --init while explicitly disabling git-lfs (to try to set up the submodule without risking any git-lfs errors)
2. if the git-submodule isn’t in the appropriate state, we perform git -C <path/to/submod> restore . to fix up the submodule (we continue to explicitly disable git-lfs)
3. Then we try to use git-lfs to try to fetch and checkout the large files (this would replace the corresponding “pointer-files” directly tracked in the repository that git-lfs maps to the large files)
4. If the previous step failed, we manually replace the pointer files.

I just need to update some of the documentation, then we will be ready to merge

Ok, this PR is ready for review.

To briefly summarize, the tools/ci-setup-submodule.py script handles all of the extra work to make sure things actually work properly. Its a little long, but most of the logic is essential (a lot of the code implements the backup strategy for when things fail). With that said, we could delete the _progress_bar function (it prints out a nice progress-bar when you run the script locally, but that doesn't seem to work in the CI at the moment)

Looks great!